Problem:

Currently, GreatExpectationsRunner uploads the HTML reports generated by evaluating an expectations suite to a unique Synapse folder for each dataset. The file name is the timestamp for when the report was generated. This poses a couple of problems, including the proliferation of files in those folders (1 per agora-data-tools run) and difficulty in finding reports of interest within the folders.

Solution:

Update agora-data-tools to instead generate report files named after the expectation suite/dataset all into one folder. We then rely on Synapse's versioning to keep track of new versions by uploading the reports with forceVersion=True. This should provide a much better interface for individuals looking to examine reports.

Notes:

• The old folders are still in Synapse. If we don't care about those past reports I can delete them, otherwise I could index them back into the new file version histories. Let me know.
• Changes were made to the configuration files. gx_folder is now a "global" configuration which points to the "Great Expectations Reports" folder for prod and testing. Datasets with existing expectation suites now have gx_enabled as part of their individual configuration. gx_enabled is now checked during data processing to determine if GX should be run against a dataset.
• Tests in test_gx.py and test_process are updated as needed.
• Documentation is updated to reflect the changes to the configuration files and GX expectation suite contribution process.
• Example report file in the testing folder.